[YUNIKORN-2884] Task fail with post allocated but the pod will keep p… #916
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #916 +/- ##
==========================================
- Coverage 68.20% 68.15% -0.05%
==========================================
Files 70 70
Lines 7621 7625 +4
==========================================
- Hits 5198 5197 -1
- Misses 2215 2218 +3
- Partials 208 210 +2 ☔ View full report in Codecov by Sentry. |
|
I don't think this is the correct approach. We're arguably making things worse by terminating the pod, as this makes viewing events on it for diagnostic purposes near impossible. I think we should be looking into ways to recover from this issue instead. |
|
Thanks @craigcondit, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ending
What is this PR for?
Application should also release task fail case, pod should be deleted, because TerminationType_STOPPED_BY_RM will also be called when task fail.
If we don't delete the pod, the pod will be stuck in pending state.
After applied this PR, the test passed locally.
Note:
I will create a e2e testing jira after this PR resolved.
And in future, i am working on retry policy when task failing, we will ajust this logic to pod pending and deallocate in core side, but until now, we need to make sure the task failed should be cleaned for applications.
What type of PR is it?
Todos
What is the Jira issue?
[YUNIKORN-2] Gang scheduling interface parametersHow should this be tested?
Screenshots (if appropriate)
Questions: